feat(btrblocks): let FSSTScheme compress binary arrays behind a sampled gate - #9580
feat(btrblocks): let FSSTScheme compress binary arrays behind a sampled gate#9580joseph-isaacs wants to merge 1 commit into
Conversation
…ed gate FSSTScheme gated on is_utf8(), but its compress path never validates UTF-8 -- it trains and compresses over the raw varbinview bytes, so the dtype gate was excluding binary columns from a scheme that already works on them. Widen matches() to accept binary. Because arbitrary binary payloads (hashes, ciphertexts, compressed blobs) often have no intra-value structure, binary columns do not go straight to sampling. FSST first trial-compresses a tiny strided sample of up to 64 values and enters scheme selection only when its code bytes beat the plain VarBin value bytes (views excluded) by more than 15%. Once past the gate, the ranked ratio comes from the compressor's standard sampling estimator via the new CascadingCompressor::estimate_by_sampling, so FSST competes on the same measurement basis as every other scheme. Measured at 100k rows (tests/varbin_scheme.rs), compressed nbytes: shared prefix 1,600,000 -> 734,685 nulls every 7th 1,647,348 -> 690,230 random 16B (hash) 1,600,000 -> 1,600,000 (gate skips FSST) random 256B 25,600,000 -> 25,600,000 (gate skips FSST) Binary lands byte-identical to the same content stored as Utf8, which confirms the dtype gate was not protecting the compress path. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will improve performance by 12.5%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | words_gather_dispatch_avx512[1024] |
9 ns | 8 ns | +12.5% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/fsst-binary-scheme (81d6bb8) with develop (e4b3421)2
Footnotes
-
54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
claude/varbin-scheme(d598e09) during the generation of this report, sodevelop(e4b3421) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
2 participants
Summary
Stacked on #9579.
FSSTSchemegated onis_utf8(), but its compress path never validates UTF-8 — it trains and compresses over the rawvarbinviewbytes, so the dtype gate was excluding binary columns from a scheme that already works on them. This PR widensmatches()to accept binary, but because arbitrary binary payloads (hashes, ciphertexts, compressed blobs) often have no intra-value structure, binary columns must first pass a cheap trial: FSST compresses a tiny strided sample of up to 64 values and enters scheme selection only when its code bytes beat the plainVarBinvalue bytes (views excluded) by more than 15%.Changes
FSSTScheme::matchesaccepts binary; for binary dtypesexpected_compression_ratioreturns aDeferredEstimate::Callbackimplementing the gate (vortex-btrblocks/src/schemes/string/fsst.rs).CascadingCompressor::estimate_by_samplingwrapper (vortex-compressor/src/compressor/sample.rs), so FSST competes on the same measurement basis as every other scheme — a gate-local ratio turned out to be incomparable with sampled scores and mis-ranked FSST against dict/varbin.vortex-btrblocks/tests/varbin_scheme.rs:fsst_binary_gateasserts FSST appears in the compression tree for structured payloads and never for random 16B/256B payloads;fsst_versus_varbin_on_identical_bytesasserts the same bytes compress byte-identically as Binary and Utf8. Measured at 100k rows: shared prefix 1,600,000 → 734,685; nulls every 7th 1,647,348 → 690,230; random payloads are untouched by FSST (gate skips).API Changes
Adds public
CascadingCompressor::estimate_by_samplingfor schemes whose deferred-callback estimates want to fall back to the standard sampled score.Generated by Claude Code